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1 Introduction 

For the probability distribution p{x) = p{X = x) of the random variable X, Tsallis entropy was 
defined in [Ij : 

S,iX)^-^pixyin,pix), (1) 

X 

with one parameter q as an extension of Shannon entropy, where ^-logarithm function is defined 
by Ing(x) = ^ ilg^ for any nonnegative real number x and q. Tsallis entropy plays an essential 
role in nonextensive statistics so that many important results have been published in mainly 
statistical physics [2]. In particular, see OS] for the recent results on Tsallis entropies. The field 
of the study on Tsallis entropy and related fields in statistical physics are often called Tsallis 
statistics or nonextensive statistical physics due to the nonextensivity Eq.Q in the below. We 
easily find that limg^i Sq{X) = Si{X) = — logp(x), since (/-logarithm converges to 

natural logarithm as q ^ 1. After the celebrated discover by Tsallis, it was shown that Tsallis 
entropy can be uniquely formulated by the generalized Shannon-Khinchin's axioms in [5l[6l[7]. 
The axiomatical characterization of Tsallis entropy was improved and that of Tsallis relative 
entropy was formulated in [8j. It is a crucial difference of the fundamental property between 
Shannon entropy and Tsallis entropy that for two independent random variables X and Y, 
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Shannon entropy Si{X) has an additivity 



Si{X xY) = Si{X) + Si{Y), (2) 

however, Tsalhs entropy Sq{X), {q ^ 1) has a pseudoadditivity : 

S,{X xY) = S,{X) + S,{Y) + (1 - q)S,{X)S,iY), (3) 

where Sq{X x Y) is Tsalhs joint entropy, which wih be defined in the below, for the direct 
product X X y of two independent random variables X and Y . The independence between 
X and Y phyisically means that there is no interaction between two systems X and Y . This 
pseudoadditivity of Tsallis entropy in Eq. ([3]) originally comes from that of g- logarithm function 
Ing(x). We pay attention whether an additivity (such as Eq.©) of entropies for two independent 
random variables holds or not (such as Eq.Q). An entropy is called a nonadditive entropy if it 
does not satisfy the additivity such as Eq.([2|), while an entropy is called an additive entropy if 
it satisfies the additivity specified in Eq.([2]). The entropy of type /3 introduced by Daroczy in 
|17j (also called the structural a-entropy introduced by Havrda and Charvat in |18j): 



is a nonadditive entropy. 

We are now interested in Tsallis entropy which is a typical nonadditive entropy. From 
Eq.©, we easily find that we have a subadditivity of Tsallis entropies for two independent 
random variables X and Y in the case of g > 1 since Tsallis entropy is nonnegative. In the first 
of the section 3, we will review a subadditivity of Tsallis entropies for general random variables 
X and Y in the case of g > 1. 

From the entropy theoretical point of view, for the related entropies such as Tsallis condi- 
tional entropy and Tsallis joint entropy were introduced in [9l [TOt [TT] and then some interesting 
relations between them were derived. In this paper, in order to derive the relation between 
them, we adopt the following definition of Tsallis conditional entropy and Tsallis joint entropy. 

Definition 1.1 For the conditional probability p{x\y) = p{X = x\Y = y) and the joint proba- 
bility p{x, y) = p{X = x,Y = y), we define Tsallis conditional entropy and Tsallis joint entropy 
by 

S,{X\Y) ^ -^p(x,y)nn,p(:E|y), {q + 1), (4) 

and 

5,(X,y)^-^p(x,2/)^ln,;5(x,y), {q + \). (5) 
We note that the above definitions were essentially introduced in [T7] by 



y 

except for the difference of the multiplicative function. And then a chain rule and a subadditivity: 

Hfs{X,Y) = H0{X) + His{Y\X), 
Hf^{Y\X) < HfsiY), /3>1, 
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were shown in Theorem 8 of [17J. In this paper, we call the entropy defined by Eq.Q, Eq.(j4|) 
or Eq.([5]) Tsallis type entropy. It is remarkable that Eq.(|4l) can be easily deformed by 

S,{X\Y) = Y,p{yrS,{X\y). (6) 

y 

Note that for g 7^ 1 we have Sq{X\Y) = Yl,yP{yYSq{X) ^ Sq{X) when X and Y are indepentent 
each other (i.e., p{x\y) = p{x) for all x and y). Eq.Q can be further regarded that the following 
expectation value : 

i 

depending on the parameter q is adopted. Since the expectation value Eq{X) has a crucial 
lack that it dose not hold Eq{l) = 1 for g 7^ 1 in general, a slightly modified definitions, the 
normalized Tsallis entropies such as 

w = — — ' ^ ^ ' 

are sometimes used [101 HI] - and then important and meaningful results in physics are derived 
for the normalized nonadditive conditional entropy defined in [GlfTO]. However, as pointed out in 
|13j . the normalized Tsallis entropies are not stable, but the original Tsallis entropies are stable 
as experimental quantities so that it is meaningful to study the original Tsallis entropies from 
the physical point of view. Inspired by these pioneering works [6l[10l[13], we therefore study the 
information theoretical properties of the original Tsallis type entropies without normalization 
in this paper. As advantages in information theoretical study, our approach shows chain rules 
and subadditivities of the Tsallis entropies and then naturally leads to define the Tsallis mutual 
entropy. Finally the rough meaning of the parameter q of the Tsallis entropy is given from the 
information theoretical viewpoint. Both entropies defined in Eq.(j3|) and Eq.Q recover the usual 
Shannon entropies as q ^ 1. 

In the past, some parametrically extended entropies were introduced in the mathematical 
point of view. Especially, the Renyi entropy jl4l[l5] (see also |16j ) 

i?,(X) = ^logj;p(x)^ (qy^l). (8) 

is famous. For Tsallis entropy and Renyi entropy, we have the following relation, 

R^{X) = -^\og{l + {l-q)Sq{X)], (g/l). (9) 
1 — q 

It is notable that Renyi entropy has an additivity: 

Rq{X>iY) = Rq{X)+Rq{Y), (10) 

for two independent random variables X and Y . Although there exsits a simple transformation 
between Tsallis entropy and Renyi entropy, they have a different structure in regards to the 
matter of their additivities. 

This paper is organized as follows. In section [2l we review a chain rule given in [T7] and give 
its generalized results. We also give the chain rule for Tsallis relative entropy. In section [3l we 
review a subadditivity given in [T7] and give its generalized results. In addition, we show several 
inequalities related Tsallis entropies. In section HI we introduce Tsallis mutual entropy and then 
give its fundamental properties. In section [5l we discuss on entropic distances and correlation 
coefficients due to Tsallis entropies. In section [6l we give some applications of Tsallis entropies 
in the context of information theory. Related remarks and discussions are given in section [71 



3 



2 Chain rules for Tsallis entropies 

It is important to study so-called a chain rule which gives the relation between a conditional 
entropy and a joint entropy in not only information theory [22] but also statistical physics. For 
these Tsallis entropies, the following chain rule holds as similar as the chain rule holds for the 
joint entropy of type /3 and the conditional entropy of type j3. 

Proposition 2.1 ffTT]) 

Sq{X,Y) = Sq{X) + Sq{Y\X). (11) 
(Therefore immediately Sq{X) < Sq{X^Y).) 

(Proof) From the formulas \\iq{xy) = In^ x + x^^*^ lug y and lug ^ = —x^~^ lug x, we have 

Sq{X,Y) = V Vp(3;,7/)lng = V (x, y) lug , \ i \ \ 

P{x,y) p{x)p{y\x) 

= Vp(x)lng— j— + V Vp(x,j/)p(x)^~Mng ^ 
= -^p(x)''lngp(x) - '^p{xy'^p{y\xyinqp{y\x) 

X X y 

= Sq{X)+Sq{Y\X). 

■ 

From the process of the proof of Proposition 12.11 we can easily find that if X and Y are 
independent (i.e., p{y\x) = p{y) for all x and y), then Eq. pTj) recovers the pseudoadditivity 
Eq.®: 

Sq{X,Y) = -'^p{xf\iiqp{x)-'^p{xf'^p{yY\D.qp{y) 

X X y 

= Sq(X) + Y,PixrSq{Y) 

X 

= Sq {X) + Sq {Y) + {l-q) Sq {X) Sq {Y) . 

As a corollary of the above Proposition 12. H we have the following lemma. 
Lemma 2.2 The following chain rules hold. 

(1) Sq{X,Y,Z) = Sq{X,Y\Z) + Sq{Z). 

(2) Sq{X,Y\Z) = Sq{X\Z) + Sq{Y\X,Z). 

(Proof) By using lug ^ = In^ y + y'^~'^ lug i, we have 

Sq (X, Y\Z ) =Sq {X, y, Z) - Sq {Z) . 

In the similar way, we have 

Sq {Y \X,Z) = Sq (X, y, Z) - Sq (X, Z) . 

Also from the Proposition 12. H we have 

Sq{XX)= Sq{X) + Sq{Y\X). 
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Therefore we have 



S^{Y\X,Z) = Sq{X,Y,Z)-Sq{X,Z) 

= Sg {X, Y\Z)+Sg (Z) - {S, {X\Z)+ Sg (Z)} 

= Sq{X,Y\Z)-Sq{X\Z). 

■ 

From (2) of of Lemma 12.21 we have 

S,{X\Z) < S,{X,Y\Z) (12) 
Remark 2.3 (2) of Lemma 12.21 can be generahzed in the foUowing formula : 

n 

Sg{Xi,---,Xn\Y) = Y,Sq{Xi\Xi_i,---,Xi,Y). (13) 

i=l 

Theorem 2.4 Let Xi, X2, ■ ■ ■ , Xn be the random variables obeying to the probability distribu- 
tion p(xi,X2, • • • , Xn). Then we have the following chain rule. 

n 

Sq {Xl,X2, • • • , Xn) = ^ Sq (Xi ■ ■ ■ ,Xi). (14) 

1=1 

(Proof) We prove the theorem by induction on n. From Proposition 12.11 we have 

Sq (-^^1,-^^2) = Sq {Xi) + Sq {X2 \Xi ) . 

Assuming that Eq. (jl4p holds for some n, from Eq. (jlip we have 

Sq {Xi, ■ ■ ■ , Xn+l) = Sq {Xi, ■ ■ ■ , Xn) + Sq {Xn+l\Xi, ■ ■ ■ , Xn) 

n 

= Sq {Xi \Xi-i, ■ ■ ■ , Xi) + Sq {Xn+l\Xn, ■ ■ ■ , Xi) , 

i=l 

which means Eq. ()14p holds for n + 1. ■ 

In the rest of this section, we discuss on the chain rule for Tsallis relative entropy. To do so, 
we use Tsallis relative entropy 

Dq{U\V)^-Y;^u{x)lnq^ (15) 

defined for two probability distributions u{x) and v{x), and q > 0. See [121 [201 ED 123] for details 
of Tsallis relative entropy. Eq. (jl5p is equivalent to the following forms. 



. , ^ y2Ju(x) - u(x)'}v(x)^''') , , 

l-q 

= Y.u{x)'^v{x)^~nnq'^^. (17) 
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Along the line of the definition of Tsallis conditional entropy Eq.Q, we naturally define 
Tsallis conditional relative entropy such as 

Dg{u{y\x)\v{y\x)) ^ V y)!"" In^ (18) 

^ v{y\x) 

= J]n(x)''t;(x)i-''J]t.(y!x)^^(yIx)i~nn,4^ (19) 



for two joint probability distributions u{x,y) and v{x,y), and two conditional probability dis- 
tributions u{y\x) and v{y\x), based on the view of Eq. (fT7|l . neither Eq. ffTSl) nor Eq. lfTHl) . Taking 
the limit as q ^ 1, Eq. ljlSp recovers the usual conditional relative entropy. Then we have the 
following chain rule of Tsallis relative entropy for general (possibly not independent) case. 

Theorem 2.5 (Chain rule for Tsallis relative entropy) 

Dg {u {x, y) \v (x, y)) = Dg {u {x) \v (x)) + Dg {u {y\x) \v {y\x)) . (20) 
(Proof) The proof follows by the direct calculations. 

^ , , , v-^ / \i "^i^iV) sr^ I \i v{x\v{y\x\ 

Dg [u X, y) \v{x,y)) = - > u x, y) In, — = - > n (x, y) In^ , . , \ . 

^\^^y) Tts u(x)u{y\x) 



u (x, y) lug ^ + In, + (1 - g) In, ^ In, 

y u(x) u[y\x) u{x) u[y\x) 



V (x) V {y |x) 

i- [1- q) m, 
V (x) ( '>J-ix)^'^~^ v{y\x 



D, (» (x) |„ (x)) - I u (x. 9) In, " > 



x,y 

Dg {u (x) \v (x)) + Dg (n (y|x) \v (y|x)) . 



u{y \x] 



Taking the limit as q ^ 1, Theorem 12.51 recovers Theorem 2.5.3 in [22j. It is known that for 
ti (y |x) = u (y) and u (y |x) = v {y), Tsallis relative entropy has a pseudoadditivity : 

Dg {u{x)u{y)\v{x)v(y)) = Dg (n(x)[t;(x))+D, (n(y)[t;(y)) + (g - 1) Dg {u{x)\v{x)) Dg {u{y)\v{y)) . 

(21) 

From the process of the above proof, we find that the chain rule Eq. (|20p recovers the pseudoad- 
ditivity Eq. (|2T]) when u (y |x) = u (y) and v (y |x) = v (y). 

3 Subadditivities for Tsallis entropies 

From Eq.Q, iov q > 1 and two independent random variables X and Y, the subadditivity holds: 

Sg{X XY)<Sg{X)+Sg{Y). 

It is known that the subadditivity for general random variables X and Y holds in the case of 
q > 1, thanks to the following proposition. 
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Proposition 3.1 ([17j) The following inequality holds for two random variables X and Y, and 

SgiX\Y) < S,iX), (22) 
with equality if and only if q = 1 and p{x\y) = p{x) for all x and y. 

(Proof) We give the similar proof given in [17] as a version of Tsallis type entropies for the 
convenience of the reader. We define the function hnq{x) = ^IZq for 9 ^ 0,g 7^ 1 and x > 0. 
Then the function Lug is nonnegative for 0<x<l,g>l and concave in x for g > 0, g ^ 1. By 
the concavity of Ln^ , we have 

^p{y)'Lnq{p{x\y)) < Lug I '^p{y)p{x\y) \ . 
y \ y / 

Taking the summation of the both sides on the above inequality on x, we have 

Y^piy)Y,^nq{pix\y)) < J]Lng(p(x)) (23) 

y X X 

Since p{y)'^ < p{y) for any y and q > 1, and Lng(t) > for any t > and g > 1, we have, 

P{yy X] Ln5(p(x|y)) < p{y) ^ Lng{p{x\y)). 

X X 

Taking the summation of the both sides on the above inequality on y, we have 

J]p(y)^^Lng(p(x|2/)) < ^^^(y) ^ Lng(p(x|y)). (24) 

y X y X 

By Eq.(l23D and Eq.dMD, we have 

y X X 

Therefore we have Sq{X\Y) < Sq{X). The equality holds when q = 1 and p{x\y) = p{x), which 
means X and Y are independent each other. Thus the proof of the proposition was completed. 



Eq. (j22p and Eq. ijlip imply the subadditivity of Tsallis entropies. 
Theorem 3.2 ([17J) For q> I, we have 

Sq{X,Y)<Sq{X)+Sq{Y). (25) 

From Eq. (j22p and Theorem l2.4l we have the following extended relation which is often called 
independence bound on entropy in information theory. 

Theorem 3.3 Let Xi, X2, • • • , X„ be the random variables obeying to the probability distribu- 
tion p{xi,X2, • • • , Xn)- Then ior q > 1, we have 

n 

Sq{Xi,---,Xn)<Y,Sli^^) (26) 

i=l 

with equality if and only if g = 1 the random variables are independent each other. 
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On the other hand, we easily find that for two independent random variables X and Y , and 
< g < 1, the superadditivity holds: 

Sq{X-KY)>Sq{X) + Sq{Y). 

However, in general the superadditivity for two correlated random variables X and Y , and 
< g < 1 does not hold. Becasue we can show many counterexamples. For example, we 
consider the following joint distribution of X and Y . 

p{xi,yi) =p(l - x),p(xi,7/2) = (1 -p)y,p{x2,yi) =px,p{x2,y2) = (l - y), (27) 

where < p,x,y < 1. Then each marginal distribution can be computed by 

p(xi) =p{l - x) + (1 -p)y,p{x2) =px + {l -p){l - y),p{yi) =p,p{y2) = l- p. (28) 

In general, we clearly see X and Y are not independent each other for the above example. Then 
the value of A = Sq{X, Y) — Sq{X) — Sq(Y) takes both positive and negative so that there does 
not exist the complete ordering between Sq{X,Y) and Sq{X) + Sq{Y) for correlated X and Y 
in the case of < g < 1. Indeed, A = -0.287089 when q = 0.8, p = 0.6, x = 0.1, y = 0.1, while 
A = 0.0562961 when q = 0.8, p = 0.6, x = 0.1, y = 0.9. 

Up to the above discussions, we have the possibility that the strong subadditivity holds in the 
case of q > 1. Indeed we can prove the following strong subadditivity for Tsallis type entropies. 

Theorem 3.4 For q > 1, the strong subadditivity 

Sq {X, Y, Z) + Sq {Z) < Sq {X , Z) + Sq {Y, Z) (29) 

holds with equality if and only \i q = 1 and, random variables X and Y are independent for a 
given random variable Z. 

(Proof) Theorem is proven by the similar way of Proposition [3Tj By the concavity of Ln^, we 
have 

^p{y\z)l.TLq{p{x\y,z)) < Luq I '^p{y\z)p{x\y,z) J 
y \ y / 

Multiplying p{zy to the both sides of the above inequality and then taking the summation on 
z and X, we have 

^p{z)''p{y\z)^Lnq{p{x\y,z)) <^p{z)''^Lnq{p{x\z)) (30) 
y,z X z X 

since J2yPiy\z)p{x\y, z) =p{x\z). 

For any y, z and g > 1, we have p{y\z)'^ < p{y\z). Then by the nonnegativity of the function 
Luq, we have 

P{y\xy ^ Lnq{p{x\y, z)) < p{y\z) ^ ljnq{p{x\y, z)). 

X X 

Multiplying p{z)'^ to the both sides of the above inequality and then taking the summation on 
y and z, we have 

^p{zfp{y\xf ^ Lng(p(x|2/, z)) < ^piz^piylz) ^ Lnq{p{x\y, z)). (31) 

y,z X y,z X 
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From Eq. (|30p and Eq. (j3ip . we have 

y,z X z X 

Thus we have 

Sq{X\Y,Z)<S^{X\Z), (32) 

which imphes 

(X, y, Z) - Sg {Y, Z) < Sg {X, Z) - Sg (Z) , 

by chain rules. The equality holds when q = 1 and p{x\y,z) = p{x\z), which means X and Y 
are independent each other for a given Z. ■ 



We find that Eq. (j29p recovers Eq. (125p . by taking the random variable Z as a trivial one. This 
means that Eq. (j29p is a generalization of Eq. (|25p . We have the further generalized inequality 
by the similar way of the proof of Theorem 13.41 

Theorem 3.5 Let Xi, ■ ■ ■ , Xn+i be the random variables. For g > 1, we have 

Sq{Xn+l\Xi, ■ ■ ■ ,Xn) < 5q , • • • ,Xn). (33) 

The subadditivity for Tsallis entropies conditioned by Z holds. 
Proposition 3.6 For g > 1, we have 

S,{X, Y\Z) < Sg{X\Z) + Sg{Y\Z). (34) 
(Proof) Summing —2Sq{Z) to the both sides in Eq. (j29p . we have 

SgiX, Y, Z) - Sg{Z) < 5,(X, Z) - 5,(Z) + 5,(y, Z) - Sq{Z). 
By chain rules, we have the proposition. ■ 
Proposition 13.61 can be generalized in the following. 

Theorem 3.7 For g > 1, we have 

Sq{Xu ■■■,Xn\Z)< Sq{Xi\Z) + ■■■ + Sq{Xn\Z). (35) 
In addition, we have the following propositions. 
Proposition 3.8 For g > 1, we have 

25, (X, y, Z) < Sg{X, Y) + S,{Y, Z) + Sq{Z, X). 
(Proof) Form Eq.(l22D and Eq.(l32]), we have 

S,{X\Y,Z)<S,{X), 

and 

S^{Z\X,Y)<Sq{Z\X). 

Thus we have 

Sq{X\Y, Z) + Sq{Z\X, Y) < S,{X) + Sg{Z\X). 

By chain rules, we have 

S,iX, Y, Z) -S^{X,Y) + Sg{X, y, Z) - Sg{Y, Z) < Sg{Z, X), 
which implies the proposition. ■ 
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Proposition 3.9 For g > 1, we have 

Sg{Xn\Xi) < Sq{X2\Xi) + • • • + Sg{Xn\Xn-l)- 

(Proof) From Eq.([TlD and Eq.dM]), we have 

Sq{Xi, ■ ■ ■ ,Xn) < Sg{Xi) + Sg{X2\Xi) + • • • Sg{Xn\Xn-l)- 

Therefore we have 

Sg{Xn\Xi) < Sg{X2, - ■ ■ ,Xn\Xi) 

= Sg{Xl,- ■ ■ ,Xn) — Sg{Xl) 

< 5'g(^2|-'^l) + • • • + 'S'q(-'^nl-'^n-l)) 



by Eq.([TT]) and Eq.([T2]). ■ 

4 Tsallis mutual entropy 

For normahzed Tsalhs entropies, the mutual information was defined in [11] with the assumption 
of its nonegativity. We define the Tsalhs mutual entropy in terms of the original {not normalized) 
Tsallis type entropies. The inequality Eq. (l22p naturally leads us to define Tsallis mutual entropy 
without the assumption of its nonegativity. 

Definition 4.1 For two random variables X and Y, and g > 1, we define the Tsallis mutual 
entropy as the difference between Tsallis entropy and Tsallis conditional entropy such that 

Ig{X;Y) = Sg{X) - Sg{X\Y). (36) 

Note that we never use the term mutual information but use mutual entropy through this paper, 
since a proper evidence of channel coding theorem for information transmission has not ever been 
shown in the context of Tsallis statistics. From Eq. ljlip . Eq. ()25p and Eq. ()22p . we easily find that 
Ig{X;Y) has the following fundamental properties. 

Proposition 4.2 (1) < Ig{X; Y) < min {Sg{X), Sg{Y)} . 

(2) Ig{X; Y) = Sg{X) + Sg{Y) - Sg{X, Y) = I g{Y ^ X) . 

Note that we have 

Sg{X) < Sg{Y) ^ Sg{X\Y) < Sg{Y\X) (37) 

from the symmetry of Tsallis mutual entropy. We also define the Tsallis conditional mutual 
entropy 

Ig{X;Y\Z) = Sg{X\Z)-Sg{X\Y,Z) (38) 

for three random variables X, Y and Z, and q > 1. 

In addition, Ig{X; Y\Z) is nonnegative, thanks to Eq. (j32p . For these quantities, we have the 
following chain rules. 

Theorem 4.3 (1) For three random variables X, Y and Z, and q > I, the chain rule holds: 

Ig{X; Y, Z) = Ig{X- Z) + Ig{X-Y\Z). (39) 
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(2) For random variables Xi, ■ ■ ■ ,X„ and Y , the chain rule holds: 

n 

Iq{Xu - ■ ■ ,Xn]Y) = Y,mXi-Y\Xi, - ■ ■ ,Xi_i). (40) 



i=l 

(Proof) (1) follows from the chain rules of Tsallis entropy. 

I,{X-Y\Z) = Sq{X\Z)-S^{X\Y,Z) 

= Sq{X\Z)-S^{X) + S,{X)-S,{X\Y,Z) 
= -I,{X;Z) + I,{X;Y,Z). 

(2) follows from the application of Eq. (jl4p and Eq. (jl3p . 

Ig {^li ' " , Xn',Y) = Sq {Xi, ■ ■ ■ , Xn) — Sq {Xi, ■ ■ ■ , Xn\Y) 

n n 
= ^ Sq {Xi • • • , Xi ) — ^ 5^ {Xi ■ ■ ■ ,Xi,Y) 

i=l i=l 

n 

= Y.Iq{Xi;Y\Xi,---,Xi_i). 

■ 

We have the following inequality for Tsallis mutual entropies by the strong subadditivity. 
Proposition 4.4 For g > 1, we have 

Iq{X;Z) <Iq{X,Y;Z). 

5 Correlation coefficients and distances for Tsallis entropies 

We discuss on the entropic distance between X and Y by means of Tsallis conditional entropy 
Sq{X\Y). If Sq{X\Y) = = Sq{Y\X), then X and Y are called equivalence and denoted by 
X = Y. Defining an entropic distance between X and Y as 

dq{X, Y) = Sq{X\Y) + Sq{Y\X), (41) 

we have the following proposition. 

Proposition 5.1 For q > 1, dq satisfies 

(i) dq{X, y) = if and only ifX = Y. 

(ii) dq{X,Y)=dq{Y,X). 

(ifi) dq{X, Z) < dq{X, Y) + dq{Y, Z) . 

(Proof) (i) and (ii) are clear from the definition of dq. In the case of n = 3 in Proposition 13.91 
we have the triangle inequality: 

Sq{X^\Xl) < Sq{X2\Xl) + Sq{X^\X2). 

Putting Xi = Z,X2 = Y and X^ = X in the aboce inequality, we have 

Sq{X\Z) < Sq{Y\Z) + Sq{X\Y). (42) 
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In addition, exchanging X and Z in the above inequality, we have 

Sq{Z\X) < S^{Y\X) + Sq{Z\Y). 

Summing the both sides of two above inequaUties, we have the proposition. ■ 

Iq{X]Y) represents a kind of correlation between X and Y. Following [25] , we define a 
parametrically extended correlation coefficient in terms of Tsallis mutual entropy such that 

for Sq{X) > 0, Sq{Y) > and q > 1. Then we have the following proposition. 
Proposition 5.2 For q > I, Sq{X) > and Sq(Y) > 0, we have the following properties. 

(i) PqiX,Y)=PqiY,X). 
(ii) 0<Pq{X,Y) < 1. 

(iii) Pq{X, y) = if and only if X and Y are independent each other and q = 1- 

(iv) pq{X, y) = 1 if and only if X = y. 

(Proof) (i) is clear from the definition oi pq. (ii) follows from < Iq{X; Y) < Sq{X) < Sq{X, Y). 
(iii) follows from Proposition EH (iv) follows from Iq{X;Y) = Sq{X,Y) ^ Sq{X\Y) = 

Sq{Y\X) = 0. 

■ 

Again, we define an entropic distance between X and Y by 

dq{X,Y)^l-pq{X,Y). (44) 

Then we have the following proposition. 

Proposition 5.3 For q > 1, Sq{X) > 0, Sq{Y) > and Sq{Z) > 0, we have the following 
properties. 

(i) dq{X, y) = if and only ifX = Y. 

(ii) dq{X,Y)=dq{Y,X). 

(iii) dqiX, Z) < dq{X, Y) + dq{Y, Z) . 

(Proof) (i) and (ii) follows from (iv) and (i) of Proposition 15.21 respectively. From chain rules 
and Eq. (jl2|) . we have 

Sq{X\Z) Sq{X\Z) 



Sq{X,Z) Sq{X\Z) + Sq{Z) 

Sq{X\Y) + Sq{Y\Z) 



< 



Sq{X\Y) + Sq{Y\Z) + Sq{Z) 

Sq{X\Y) , Sq{Y\Z) 



+ 



Sq{X\Y) + Sq{Y\Z) + Sq{Z) Sq{X\Y) + Sq{Y\Z) + Sq{Z) 
^ Sq{X\Y) ^ Sq{Y\Z) 



Sq{X\Y) + Sq{Z) Sq{Y\Z) + Sq{Z) 

(45) 



Sq{X\Y) , Sq{Y\Z) 



Sq{X,Y) Sq{Y,Z)' 
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Exchanging X and Z, we have 

S,{Z\X) ^ Sg{Z\Y) Sg{Y\X) 

S,{Z,X) - S,{Z,Y) Sg{Y,X)- ^ ' 

Summing the both sides of two inequahties Eq. (j45p and Eq. (j46p . we have (iii), since Tsalhs joint 
entropy is symmetry. 

■ 

We also define another correlation coefficient in terms of Tsallis mutual entropy by 

^ max{s[{X)X{Y)} ^^^^ 
for Sq{X) > 0, Sq{Y) > and q > 1. See [26]. Then we have the following proposition. 

Proposition 5.4 For q > I, Sq{X) > and Sq(Y) > 0, we have the following properties. 

(i) Pq{X,Y)=Pq{Y,X). 
(ii) 0<Pq{X,Y) < 1. 

(iii) Pq{X, y) = if and only if X and Y are independent each other and q = 1- 

(iv) pq{X, y) = 1 if and only iiX = Y. 
(Proof) (i)-(iv) follow straightforwardly. 

■ 

We also define an entropic distance between X and Y by 

dq{X,Y)^l-pq{X,Y). (48) 

Then we have the following proposition. 

Proposition 5.5 For q > 1, Sq{X) > 0, Sq{Y) > and Sq{Z) > 0, we have the following 
properties. 

(i) dq{X, y) = if and only if X = y. 

(ii) dq{X,Y)=dq{Y,X). 

(iii) dq{X, Z) < dq{X, Y) + dq{Y, Z). 

(Proof) (i) and (ii) follows from Proposition 15. 4i (iii) is proven by the similar way in [26]. In 
order to show (iii), we can assume Sq{Z) < Sq{X) without loss of generality. Then we prove 
(iii) in the following three cases. 

(a) Sq(Y) < Sq{Z) < Sq{X) I From Eq.^T^ and Eq.^, we have 

~ 5^(^ < g,(^[>^) , ^,(^1^) ^ Sq{X\Y) Sq{Y\Z) 

' ' Sq{X) - Sq{X) ^ Sq{X) " Sq{X) ^ Sq{Z) 

Sq(X\Y) Sq(Z\Y) - , ^ , 

(b) Sq{Z) < Sq{X) < Sq{Y) : It is shown as similar as (a). 
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(c) SgiY) < Sq{Z) < Sg{X) : From Eq.^, we have 



SgiX) V SgiX) J - SgiY) \ 5g (X) 

SgjY) - SgjX) + Sg{X\Z) ^ SgjY) - SgjX) + Sg{X\Y) + Sg{Y\Z) 

Sg{Y) - SgiY) 

dgiX,Y)+dgiY,Z). 



6 Applications of Tsallis entropies 
6.1 Generalized Fano's inequality 

As an apphcation of Tsalhs conditional entropy, we give a generalized Fano's inequality. 

Theorem 6.1 For q > 1 and two random variables taking values in the finite (alphabet) set X, 
the following inequality holds. 

SgiX\Y) < PiX / Y) \ngi\X\ - 1) + SgiPiX ^ Y)), 

where Sg represents the Tsallis binary entropy: 

Sqip) = -p^'lngp- il-p)nngil-p). 

(Proof) Define the random variables Z by 



Z 



if X = y 

1 if X^Y. 



By chain rules, SgiZ\X, Y) = and SgiZ\Y) < SgiZ) = Sg(P(X ^ Y)), we then have 

SgiX\Y) = SgiX\Y) + SgiZ\X,Y) 
= SgiX,Z\Y) 
= SgiX\Y,Z) + SgiZ\Y) 

< SgiX\Y,Z)+SgiZ) 

= SgiX\Y = y,Z = 0) + SgiX\Y = y,Z = l) + SgiPiX + Y)) 

= SgiX\Y = y,Z = 1) + SgiPiX ^Y)) 

< IngiX y^Y) + SgiPiX ^Y)), 

since the nonnegativity of Tsallis relative entropy implies — Y17=iPi^^qPi — 1%^- 



6.2 Entropy rate for Tsallis entropy 

Moreover, we discuss on the Tsallis entropy rate. 

Definition 6.2 The Tsallis entropy rate of a stochastic process X = {Xi} is defined by 

Sg (X) = lim 



Sq (Xi , • • • , X„ 



n 

if the limit exists. 
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Sq (X) can be considered as Tsallis entropy per a symbol in the limit. 

Theorem 6.3 For q > 1 and a stationary stochastic process, the Tsallis entropy rate exsits and 
is expressed by 

5JX)= hm Sq{Xn\Xi,---,Xrr-l). 

(Proof) 

By the stationary process and the inequality Eq. (j33|) . we have 



Sq {Xn \Xi, • • • , Xn-l ) — Sq {Xn+1 \X2, • • • , Xn ) 

> '5(3 (-'^n+l 1^1,-^2, •■■ j-^'^n) • 

Thus Sq {Xn \Xi, • • • , Xn-i ) Converges to a certain value s, since it is a monotone decreasing 
sequence with respect to n. That is, 

lim Sq {Xn \Xi, ■ ■ ■ ,Xn-l) = S. 

Moreover by the chain rule and Cesaro's theorem (e.g., see pp.64 of [22]), we have 
S^(X) ^ lim S,{Xi,---,Xn) ^ y Sq{X,\X,,...,X,.,) ^ ^ 

4=1 



6.3 Generalized data processing inequality 

Finally we prove the data processing inequality for Tsallis mutual entropy. Let X, Y and Z be 
random variables. If the conditional probability distribution of the random variable Z depends 
only on the random variable 1", namely the random variable Z is independent of the random 
variable X, then it is called that X, Y and Z form a simple Markov chain, and denoted by 
X ^Y ^ Z. 

Theorem 6.4 \i X ^Y ^ Z, then we have the inequality 

Iq{X-Z)<Iq{X;Y) (49) 
for (? > 1, with equality if and only if X — )• Z — )• y. 

(Proof) From Eq. (|39|) . Iq{X\Y, \Z) can be expanded in the following two ways. 

Iq{X-Y)+Iq{X-Z\Y)=Iq{X-Y,Z)=Iq{X-Z)+Iq{X-Y\Z). (50) 

Since the nonnegativity Iq {X;Y \Z) and Iq {X; Z \Y) = because of the Markovity, we have 
Iq{X;Z) < Iq{X;Y). If Iq{X;Y\Z) = 0, namely X ^ Z ^ Y, then we have Iq{X;Z) = 
Iq{X-Y). 

■ 

We easily find that the following corollary holds. 

Corollary 6.5 (1) If Z = f{Y) with a certain mapping /, then Iq{X; f{Y)) < Iq{X; Y), with 
equality if and only if / is a one-to-one mapping. 

(2) If X ^ y ^ Z, then Iq{X; Y\Z) < Iq{X; Y). 
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7 Remarks and discussions 

7.1 Chain rule and subadditivity for nonadditive entropies 

In [7], the nonadditive entropies including Tsallis entropy and the entropy of type /3 was ax- 
iomatically characterized by the generahzed Shannon-Khinchin's axiom in the fohowing manner. 
The function Sg{xi, ■ ■ ■ , x„) is assumed to be defined for the n-tuple (xi, • • • , x„) belonging to 



A„ = {pi, ■ ■ ■ ,Pn)\ '^Pi = '^^Pi > = 1,2, • • • ,n) 

i=l 



and to take a value in = [0,oo). Then it was shown in [7] that four conditions continuity, 
maximality, generalized Shannon additivity and expandability in [7] determine the function Sg : 
An —7- R"^ such that 

Sq{xi, ■ ■ ■ , Xn) = - Tr\ ' {^^) 

<p{q) 

where (p{q) is characterized by the following conditions: 

(i) (j){q) is continuous and 0(g) (1 — q) > for q ^ 1. 

(ii) limq^i (p{q) = and (p{q) 7^ for q 7^ 1. 

(iii) There exists (a, b) C R^ such that a < 1 < 6 and 0(g) is differentiable on (a, 1) and (1, b). 

(iv) There exists a positive constant number k such that limg_j.i ~ 
Simple calculations show the nonextensivity 

Sg{X XY)= Sg{X) + Sg{Y) + ( ^ ) (X ) (F ) 

for the nonadditive entropy Sq. From the view of the generalization Eq. (|5ip . we have the 
following chain rule. 

Lemma 7.1 For the nonadditive entropy 

and any continuous function (f){q), we have the chain rule 

Sg{X,Y)=Sq{X)+Sq{Y\X), 

where the nonadditive joint entropy and the nonadditive conditional entropy are defined by 

Ex,y (Pix, vY - P{x, y)) 

and 

Sg{X\Y) ^ Y.P(yySgiX\y) = ^^Piv) 
y y 

respectively. 
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(Proof) It follows immediately. 

S,{Y\X) = 



m 

Ex,y jpj^^ vY - pj^^ y)) ipj^)" - pjx)) 

S,{X,Y)-Sq{X). 



Prom the view of the generalization Eq. (j5ip . Proposition 13.11 also generalized as follows. 

Lemma 7.2 For the nonadditive entropy Sq{X) where 4){q) is continuous and 0((?)(1 — > 
for g 7^ 1, we have the inequality : 

Sq{X\Y) < Sg{X). 

(Proof) Since the function Lnq{x) = ^^^,{q > 0, g 7^ 1,x > 0) is nonnegative for < 
X < l,q > 1 and concave in x for q > 0,q 1, the theorem follows as similar as the proof of 
Proposition 13. li 

■ 

Form Lemma |7. II and Lemma 17.21 we have the subadditivity for nonadditive entropy. 

Theorem 7.3 (Subadditivity for nonadditive entropy) For the nonadditive entropy Sq{X) 
where (j){q) is continuous and (/>(g)(l — > for g 7^ 1, we have the subadditivity : 

Sq{X,Y)<Sq{X)+Sq{Y). 

We note that we need the condition that 0((?)(1 — q) > for g 7^ 1 to prove Lemma 17.21 
while we do not need any condition to prove Lemma |7. II 



7.2 Inequalities on pseudoadditivity 

The pseudoadditivity Eq. ([3]) for independent random variables X and Y gives rise to the mathe- 
matical interest whether we have the complete ordering between the left hand side and the right 
hand side in Eq.© for two general random variables X and Y. Such a kind of problem was 
taken in the paper [25] for the normalized Tsallis type entropies which are different from the 
definitions of the Tsallis type entropies in the present paper. However, its inequality appeared 
in (3.5) of the paper [28] was not true as we found the counter example in [29j. 

Unfortunately, in the present case, we also find the counter example for the inequalities 
between Sq{X,Y) and Sq{X) + Sq{Y) + (1 - q)Sq{X)Sq{Y). In the same setting of Eq.dlZD 
and Eq.dSHD, S = Sq{X,Y) - {Sq{X) + Sq{Y) + (1 - q)Sq{X)Sq{Y)} takes both positive and 
negative values for both cases < q < 1 and q > 1. Indeed, 6 = 0.00846651 when q = 
1.8,p = 0.1, X = 0.1, y = 0.8, while 6 = -0.0118812 when q = 1.8,p = 0.1, x = 0.8, y = 0.1. 
Also, 6 = 0.00399069 when q = 0.8, p = 0.1, x = 0.8, y = 0.1, while 6 = -0.0128179 when 
q = 0.8, p = 0.1, X = 0.1,2/ = 0.8. 

Therefore there does not exist the complete ordering between Sq{X, Y) and Sq{X) + Sq{Y) + 
(1 - q)Sq{X)Sq(Y) for both cases < g < 1 and g > 1. 
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7.3 A remarkable inequality derived from subadditivity for Tsallis entropies 

From Eq. (|25p . we have the following inequality 



n I m \ m / n 



E E^.- +E E^^. ^EE^^+ EE^d (52) 

for r > 1 and pij satisfying < pij < 1 and 'l2T=iPij — 1- -^y putting pij = — — - 

in Eq. (j52p , we have the following inequality as a corollary of Theorem 13.21 

Corollary 7.4 For r > 1 and Ojj > 0, 

n / m \ m / n \^ n m / n m \ 

E E«^0 +E E«^0 ^EE4+ EE«^0 • (53) 

It is remarkable that the following inequality holds [27j 

n / m \ n m 

E E«.- ^EE«^- (54) 

for r > 1 and Uij > 0. 



7.4 Difference between Tsallis entropy and Shannon entropy 

We point out on the difference between Tsallis entropy and Shannon entropy from the viewpoint 
of mutual entropy. In the case of g = 1, the relative entropy between the joint probability p{x, y) 
and the direct probability p{x)p{y) is equal to the mutual entropy: 

Di{{X,Y)\X xY) = Si{X) - Si{X\Y). 

However, in the general case {q ^ 1), there exists the following relation: 

D,{iX,Y)\X xY) = SgiX) - Sg{X\Y) 

+ '^p{xi,yj) {p{xiy~^lnqp{xi) +p[yjY'^\nqp{yj) - p{xi,yjy~^ lnqp{xi)p{yj)} , (55) 

which gives the crucial difference between the special case {q = 1) and the general case {q ^ 1). 
The third term of the right hand side in the above equation Eq. (|55p vanishes if g = 1. The 
existence of the third term of Eq. (j55p means that we have two possibilities of the definition of 
Tsallis mutual entropy, that is, Iq{X; Y) = Sq{X) - Sq{X\Y) or Iq{X; Y) = Dq{{X, Y)\X x Y). 
We have adopted the former definition in the present paper, along with the definition of the 
capacity in the origin of information theory by Shannon [21] . 



7.5 Another candidate of Tsallis conditional entropy 

It is remarkable that Tsallis entropy Sq{X) can be regarded as the expected value of luq 
that is, since Ing(x) = — x^"'^ lnq(l/x), it is expressed by 

5,(X) = ^p(x)ln,-ly, (g^l), (56) 

where the convention Olnq(-) = is set. Along with the view of Eq. (|56p . we may define Tsallis 
conditional entropy and Tsallis joint entropy in the following. 
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Definition 7.5 For the conditional probability p{x\y) and the joint probability p(x, y), we define 
Tsallis conditional entropy and Tsallis joint entropy by 



We should note that Tsallis conditional entropy defined in Eq. (j57p is not equal to that defined 
in Eq.@, while Tsallis joint entropy defined in Eq. ()58p is equal to that defined in Eq.([5]). If we 
adopt the above definitions Eq. ([57|) instead of Eq.Q, we have the following inequality. 

Proposition 7.6 For g > 1, we have 



construct a parametrically extended information theory in this case. 

8 Conclusion 

As we have seen, we have found that the chain rules for the Tsallis type entropies hold as similar 
way of the proofs of the entropy of type /3. This result was generalized from a few insights. Also 
we have proved the strong subadditivity of the Tsallis type entropies for the general two random 
variables in the case oi q> 1. Moreover it has been shown that in general the superadditivity of 
Tsallis entropies does not hold in the case of < g < 1. Thus we could give important results for 
Tsallis entropies in the case of g > 1 from the information theoretical point of view. Therefore 
we have the possibility that the parametrically extended information theory will be constructed 
by starting from Tsallis entropy for q > 1. In other words, there is less possibility of such a 
construction in the case of < (; < 1 if our stage still in the usual probability theory, because 
the relation which should be satisfied in the contrast to the usual information theory does not 
hold in the case of < g < 1. This gives us the rough meaning of the parameter q. That is, we 
found the crucial different results between Sq{X) for g > 1 and Sq{X) for < g < 1 from the 
information theoretical point of view. For g > 1, we showed that the Tsallis type entropies have 
the similar results to Shannon entropies. In other words, it has been shown that the definition 
of Tsallis conditional entropy Eq.(j3|) and Tsallis mutual entropy Eq. (|36p have a possibility to be 
a pioneer to construct a nonadditive information theory. 

Our results in the present paper are fundamental so that we are convinced that these results 
will help the progresses of nonextensive statistical physics and information theory. We are now 
studying the coding problem in nonadditive information theory and searching for the precise 
meaning of the parameter q. 
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